Automatic Discovery and Transfer of Task Hierarchies in Reinforcement Learning

strategy games offer special challenges and opportunities for studying transfer learning. These domains are complex, and good performance requires selecting long chains of actions to achieve subgoals needed for ultimate success. Reinforcement learning in these domains, because it involves a process of exploratory trial and error, can take a very long time to discover these long action chains. Fortunately, it is often possible to study smaller versions of these domains that share the same fundamental structure but that involve fewer objects and smaller state spaces. Reinforcement learning on these smaller domains is much faster. If it can discover the shared structure and transfer it to the large-scale domains, then this provides a much more efficient way of achieving good performance. Our approach is based on the claim that the key to transfer learning is to discover and represent deep forms of knowledge that are invariant across multiple domains. Consider the problem of driving to work. There are surface aspects, such as the amount of time it takes to get from home to the office or the selection of the best route, that may be highly regular, but they are unlikely to transfer when you move to a new city. On the other hand, the task structure involved in driving, such as starting the car, driving, obeying traffic laws, parking, and so on, depends only on the causal structure of the actions involved, and hence transfers more successfully from one city to another. We are interested in transferring task knowledge between source and target domains that share the same causal structure, that is, the actions in both domains depend upon and influence the same state variables. This is weaker than assuming that the behavior of actions is exactly identical in two domains. For Articles